In computer display systems, it may be desirable to project images that will be viewed by a user onto arbitrary surfaces. For example, in flight simulators, it may be desirable to project computer-generated images of scenes that would be viewed by a pilot onto a display screen. The display screen on which the images are projected may have a conical shape to simulate the cockpit of an aircraft and so that the user will be immersed in the flight simulation environment. In order to correctly display each pixel in the projected image, the projector must be calibrated according to the depth of each point on the display screen so that a two-dimensional source image can be pre-distorted for undistorted display on the three-dimensional display screen. Requiring that the projector be specially calibrated to a particular display screen makes the projector and the flight simulation program unsuitable for use in arbitrary environments. For example, it may be desirable to conduct a flight simulation in an arbitrary location, such as an office in which the walls form a parallelepiped structure. However, because immersive flight simulators are calibrated to their particular display screens, such simulators are unsuitable for use in different environments. Accordingly, portable flight simulators are limited to display on standard two-dimensional LCD or CRT display screens, which greatly decreases their utility in training pilots.
Another example for which it may be desirable to display projected images on arbitrary surfaces occurs in standard office environments. In this example, a user may desire to project the user interface associated with a computer program, such as an email program, on a wall of the user's office. If the wall forms a planar surface, a standard projector can be used, and distortion in the projected image should be minimal. However, if the user desires that the display span more than one wall, a portion of a wall and the ceiling or the floor, or any other non-planar surface, portions of the image may be distorted due to the points on which individual pixels are displayed being located at different distances from the projector. Accordingly, the user or the projector manufacturer will be required to alter the projector and/or the display software to accommodate the different surfaces. Requiring projector customization for each display surface is impractical and undesirable in light of the number of potential variations in display surfaces.
In light of the problems associated with displaying images on non-planar surfaces, it may be desirable to acquire depth information regarding the surfaces and to pre-distort the images for undistorted display on the non-planar surfaces. One method that has been used to obtain depth information is referred to as structured light depth extraction. In structured light depth extraction, a projector projects patterns of stripes or other images onto a surface. A camera detects the patterns as they are reflected from the surface. Depth extraction software is programmed with the location of the projector and the camera and computes the depth of each point in the image based on translations in locations of pixels from the projected image to the reflected image. Thus, structured light depth extraction can be used to detect the geometry of a display surface and the depth information can be used to pre-distort the image so that the displayed image will appear undistorted on non-planar surfaces. However, one problem with using structured light in combination with projected user images is that the structured light patterns are not visually pleasing to users. For example, a user who wants to view a projection of his or her email interface on one or more walls of the user's office will not want the email interface to be projected simultaneously with a visible striped pattern, as the pattern would impair viewing of the email interface.
In light of the competing goals of obtaining depth information and providing an environment that is visually pleasing to the user, structured light patterns have been generated using non-visible wavelengths of light. In one method, infrared light is used to project structured light patterns. For example, an infrared projector may be used to project structured light patterns onto an object being imaged. The infrared images may be collected and used to compute depth information for the object. A depth-corrected image of the object may then be displayed to the user. Because the infrared patterns are outside of the visible light wavelength, the end user does not perceive the patterns. However, one problem with using infrared structured light patterns is that an infrared projector is required. Infrared projectors are more expensive than visible light projectors and are not universally available. In systems where it is desirable to project a user image simultaneously with the projection of infrared structured light patterns, two projectors would be required—an infrared projector for the structured light patterns and a visible light projector for the user image. Requiring two projectors further increases the expense and decreases the reliability of the display system. Another problem with using infrared projectors is that current infrared projectors are not capable of projecting dynamically changing patterns at speeds that are sufficiently fast for simultaneous depth extraction and display.
Thus, in light of the difficulties associated with conventional projection display and structured light depth extraction systems, there exists a need for improved methods, systems, and computer program products for imperceptibly embedding structured light patterns in projected color images for display on planar and non-planar surfaces.